ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the "pessimistic" approach by Fawcett (2006).Comment: 16 pages, 3 figures, cod

ciftiTools: A package for reading, writing, visualizing and manipulating CIFTI files in R

Surface- and grayordinate-based analysis of MR data has well-recognized advantages, including improved whole-cortex visualization, the ability to perform surface smoothing to avoid issues associated with volumetric smoothing, improved inter-subject alignment, and reduced dimensionality. The CIFTI grayordinate file format introduced by the Human Connectome Project further advances grayordinate-based analysis by combining gray matter data from the left and right cortical hemispheres with gray matter data from the subcortex and cerebellum into a single file. Analyses performed in grayordinate space are well-suited to leverage information shared across the brain and across subjects through both traditional analysis techniques and more advanced statistical methods, including Bayesian methods. The R statistical environment facilitates use of advanced statistical techniques, yet little support for grayordinates analysis has been previously available in R. Indeed, few comprehensive programmatic tools for working with CIFTI files have been available in any language. Here, we present the ciftiTools R package, which provides a unified environment for reading, writing, visualizing, and manipulating CIFTI files and related data formats. We illustrate ciftiTools' convenient and user-friendly suite of tools for working with grayordinates and surface geometry data in R, and we describe how ciftiTools is being utilized to advance the statistical analysis of grayordinate-based functional MRI data.Comment: 41 pages, 6 figure

Relating multi-sequence longitudinal intensity profiles and clinical covariates in new multiple sclerosis lesions

Structural magnetic resonance imaging (MRI) can be used to detect lesions in the brains of multiple sclerosis (MS) patients. The formation of these lesions is a complex process involving inflammation, tissue damage, and tissue repair, all of which are visible on MRI. Here we characterize the lesion formation process on longitudinal, multi-sequence structural MRI from 34 MS patients and relate the longitudinal changes we observe within lesions to therapeutic interventions. In this article, we first outline a pipeline to extract voxel level, multi-sequence longitudinal profiles from four MRI sequences within lesion tissue. We then propose two models to relate clinical covariates to the longitudinal profiles. The first model is a principal component analysis (PCA) regression model, which collapses the information from all four profiles into a scalar value. We find that the score on the first PC identifies areas of slow, long-term intensity changes within the lesion at a voxel level, as validated by two experienced clinicians, a neuroradiologist and a neurologist. On a quality scale of 1 to 4 (4 being the highest) the neuroradiologist gave the score on the first PC a median rating of 4 (95% CI: [4,4]), and the neurologist gave it a median rating of 3 (95% CI: [3,3]). In the PCA regression model, we find that treatment with disease modifying therapies (p-value < 0.01), steroids (p-value < 0.01), and being closer to the boundary of abnormal signal intensity (p-value < 0.01) are associated with a return of a voxel to intensity values closer to that of normal-appearing tissue. The second model is a function-on-scalar regression, which allows for assessment of the individual time points at which the covariates are associated with the profiles. In the function-on-scalar regression both age and distance to the boundary were found to have a statistically significant association with the profiles

ANALYTIC PROGRAMMING WITH fMRI DATA: A QUICK-START GUIDE FOR STATISTICIANS USING R

Functional magnetic resonance imaging (fMRI) is a thriving field that plays an important role in medical imaging analysis, biological and neuroscience research and practice. This manuscript gives a didactic introduction to the statistical analysis of fMRI data using the R project along with the relevant R code. The goal is to give tatisticians who would like to pursue research in this area a quick start for programming with fMRI data along with the available data visualization tools

Removing inter-subject technical variability in magnetic resonance imaging studies

Magnetic resonance imaging (MRI) intensities are acquired in arbitrary units, making scans non-comparable across sites and between subjects. Intensity normalization is a first step for the improvement of comparability of the images across subjects. However, we show that unwanted inter-scan variability associated with imaging site, scanner effect and other technical artifacts is still present after standard intensity normalization in large multi-site neuroimaging studies. We propose RAVEL (Removal of Artificial Voxel Effect by Linear regression), a tool to remove residual technical variability after intensity normalization. As proposed by SVA and RUV [Leek and Storey, 2007, 2008, Gagnon-Bartsch and Speed, 2012], two batch effect correction tools largely used in genomics, we decompose the voxel intensities of images registered to a template into a biological component and an unwanted variation component. The unwanted variation component is estimated from a control region obtained from the cerebrospinal fluid (CSF), where intensities are known to be unassociated with disease status and other clinical covariates. We perform a singular value decomposition (SVD) of the control voxels to estimate factors of unwanted variation. We then estimate the unwanted factors using linear regression for every voxel of the brain and take the residuals as the RAVEL-corrected intensities. We assess the performance of RAVEL using T1-weighted (T1-w) images from more than 900 subjects with Alzheimer’s disease (AD) and mild cognitive impairment (MCI), as well as healthy controls from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. We compare RAVEL to intensity-normalization-only methods, histogram matching, and White Stripe. We show that RAVEL performs best at improving the replicability of the brain regions that are empirically found to be most associated with AD, and that these regions are significantly more present in structures impacted by AD (hippocampus, amygdala, parahippocampal gyrus, enthorinal area and fornix stria terminals). In addition, we show that the RAVEL-corrected intensities have the best performance in distinguishing between MCI subjects and healthy subjects by using the mean hippocampal intensity (AUC=67%), a marked improvement compared to results from intensity normalization alone (AUC=63% and 59% for histogram matching and White Stripe, respectively). RAVEL is generalizable to many imaging modalities, and shows promise for longitudinal studies. Additionally, because the choice of the control region is left to the user, RAVEL can be applied in studies of many brain disorders

Thrombolytic removal of intraventricular haemorrhage in treatment of severe stroke: results of the randomised, multicentre, multiregion, placebo-controlled CLEAR III trial

Background: Intraventricular haemorrhage is a subtype of intracerebral haemorrhage, with 50% mortality and serious disability for survivors. We aimed to test whether attempting to remove intraventricular haemorrhage with alteplase versus saline irrigation improved functional outcome. Methods: In this randomised, double-blinded, placebo-controlled, multiregional trial (CLEAR III), participants with a routinely placed extraventricular drain, in the intensive care unit with stable, non-traumatic intracerebral haemorrhage volume less than 30 mL, intraventricular haemorrhage obstructing the 3rd or 4th ventricles, and no underlying pathology were adaptively randomly assigned (1:1), via a web-based system to receive up to 12 doses, 8 h apart of 1 mg of alteplase or 0·9% saline via the extraventricular drain. The treating physician, clinical research staff, and participants were masked to treatment assignment. CT scans were obtained every 24 h throughout dosing. The primary efficacy outcome was good functional outcome, defined as a modified Rankin Scale score (mRS) of 3 or less at 180 days per central adjudication by blinded evaluators. This study is registered with ClinicalTrials.gov, NCT00784134. Findings: Between Sept 18, 2009, and Jan 13, 2015, 500 patients were randomised: 249 to the alteplase group and 251 to the saline group. 180-day follow-up data were available for analysis from 246 of 249 participants in the alteplase group and 245 of 251 participants in the placebo group. The primary efficacy outcome was similar in each group (good outcome in alteplase group 48% vs saline 45%; risk ratio [RR] 1·06 [95% CI 0·88–1·28; p=0·554]). A difference of 3·5% (RR 1·08 [95% CI 0·90–1·29], p=0·420) was found after adjustment for intraventricular haemorrhage size and thalamic intracerebral haemorrhage. At 180 days, the treatment group had lower case fatality (46 [18%] vs saline 73 [29%], hazard ratio 0·60 [95% CI 0·41–0·86], p=0·006), but a greater proportion with mRS 5 (42 [17%] vs 21 [9%]; RR 1·99 [95% CI 1·22–3·26], p=0·007). Ventriculitis (17 [7%] alteplase vs 31 [12%] saline; RR 0·55 [95% CI 0·31–0·97], p=0·048) and serious adverse events (114 [46%] alteplase vs 151 [60%] saline; RR 0·76 [95% CI 0·64–0·90], p=0·002) were less frequent with alteplase treatment. Symptomatic bleeding (six [2%] in the alteplase group vs five [2%] in the saline group; RR 1·21 [95% CI 0·37–3·91], p=0·771) was similar. Interpretation: In patients with intraventricular haemorrhage and a routine extraventricular drain, irrigation with alteplase did not substantially improve functional outcomes at the mRS 3 cutoff compared with irrigation with saline. Protocol-based use of alteplase with extraventricular drain seems safe. Future investigation is needed to determine whether a greater frequency of complete intraventricular haemorrhage removal via alteplase produces gains in functional status

ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI

Ischemic stroke is the most common cerebrovascular disease, and its diagnosis, treatment, and study relies on non-invasive imaging. Algorithms for stroke lesion segmentation from magnetic resonance imaging (MRI) volumes are intensely researched, but the reported results are largely incomparable due to different datasets and evaluation schemes. We approached this urgent problem of comparability with the Ischemic Stroke Lesion Segmentation (ISLES) challenge organized in conjunction with the MICCAI 2015 conference. In this paper we propose a common evaluation framework, describe the publicly available datasets, and present the results of the two sub-challenges: Sub-Acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES). A total of 16 research groups participated with a wide range of state-of-the-art automatic segmentation algorithms. A thorough analysis of the obtained data enables a critical evaluation of the current state-of-the-art, recommendations for further developments, and the identification of remaining challenges. The segmentation of acute perfusion lesions addressed in SPES was found to be feasible. However, algorithms applied to sub-acute lesion segmentation in SISS still lack accuracy. Overall, no algorithmic characteristic of any method was found to perform superior to the others. Instead, the characteristics of stroke lesion appearances, their evolution, and the observed challenges should be studied in detail. The annotated ISLES image datasets continue to be publicly available through an online evaluation system to serve as an ongoing benchmarking resource (www.isles-challenge.org).Peer reviewe